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13,  ABSTRACT  (Msxiwum  2Q0'MOfas) 

We  are  working  on  an  enabling  technology  to  facilitate  the  development  of  physiological 
indices  of  mental  workload  that  could  be  used  in  high  performance  aircraft.  To  date,  we 
have  designed  and  implemented  the  core  components  of  a  neural-network  based 
algorithm  for  deriving  continuous  mental  workload  indices  from  continuous  recordings 
of  brain,  scalp  muscle,  eye  and  heart  electrical  activity.  We  have  also  designed  an 
experiment  to  test  the  adequacy  of  this  algorithm,  and  have  developed  technologies  to 
perform  the  experiment  including:  (1)  designing  a  task  battery  to  initially  test  the  ability 
of  the  network  algorithm  to  generalize  across  cognitive  functions  relevant  to  piloting 
aircraft;  and  (2)  implementing  a  software  library  that  could  be  used  to  efficiently  present 
the  task  stimuli  using  the  same  personal  computer  which  also  collects  32  channels  of 
electrophysiological  data.  We  have  tested  the  integrated  system  and  have  found  it 
capable  of  providing  accurate  timing  of  task  stimuli,  subject  responses,  and 
electrophysiological  data. 
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I.  BACKGROUND 

We  proposed  to  build  a  data  collection  and  software  workbench,  called  the  Workload 
Assessment  Research  workstation  (WAR  Station),  to  facilitate  research  on  mental 
workload.  The  WAR  Station  is  to  include  hardware  and  software  for  collecting 
electrophysiological  and  behavioral  data  associated  with  tasks  used  in  mental  workload 
research  and  a  collection  of  software  tools  for  analyzing  these  data  with  the  goal  of 
building  and  testing  practical  mental  workload  indices.  The  indices  would  be  bas^  on  a 
particular  neural-network  analytic  paradigm  used  in  our  prior  research  (Gevins,  1980; 
Gevins  and  Morgan,  1986,  1988).  The  algorithm  would  greatly  simplify  the  use  of 
neural  networks  for  mental  workload  researchers  since  it  automatically  generates  a 
neural  network  and  selects  optimal  variables  for  classification. 

Much  of  this  system  builds  on  hardware  and  software  under  development  in  other 
projects  at  SAM  Technology.  The  major  work  required  of  this  project  includes 

1)  designing,  implementing,  and  testing  a  novel  neural  network  algorithm  using  a  few 
techniques  from  the  algorithm  used  in  our  prior  research  and  current  multilayer 
perceptron  techniques; 

2)  performing  an  experiment  to  provide  an  initial  test  of  the  utility  of  our  analytic 
approach  for  buildmg  practical  indices  of  mental  workload; 

3)  building  the  WAR  Station  by  integrating  workload-specific  software  with  software  and 
hardware  components  built  under  other  projects  at  SAM  Technology;  and 

4)  surveying  Air  Force  researchers  to  determine  desirable  system  features  and  work  out 
what  data  import  capabilities  will  be  needed. 

These  four  items  reflect  an  approved  revision  of  our  statement  of  work  which  consisted 
of  appending  the  second  item  to  our  work  plan  and  postponing  integration  of  our  system 
with  commercial  moving-base  flight  simulators  ' 


n.  SUMMARY 
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Because  full  development  of  the  WAR  Station  depends  on  developments  of  other 
projects  being  performed  at  SAM  Technology,  it  is  most  efficient  to  stage  in  work  on  the 
WAR  Station  project  so  that  the  maximum  effort  takes  place  when  the  prerequisite 
technologies  become  available.  At  this  time,  we  have  completed  one  section  of  the  neural 
network  algorithm,  designed  a  workload  experiment  to  test  our  analysis  approach. 
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developed  tools  for  implementing  neurocognitive  tasks  that  can  be  used  to  manipulate 
mental  workload,  and  integrated  a  recently  completed  electrophysiological  recording 
system  with  task  presentation  software.  These  efforts  reflect  about  six  person  months  of 
effort  charged  to  this  project 

m.  PROGRESS  REPORT 

Neural  Network  Algorithm 

An  extension  to  the  Viglione- Joseph  neural  network  (Joseph,  1961;  Viglione,  1970; 
Gevins,  1980;  Gevins  and  Morgan,  1986,  1988)  to  handle  more  than  two-class  pattern 
recognition  problems  was  derived  then  implemented  using  the  C+-t-  programming 
language.  Presendy,  the  algorithm  automatically  creates  a  network  that  is,  effectively,  a 
two  layered  network  with  lateral  inhibition  between  second  layer  neural  units.  The  first 
layer  consists  of  neural  units  that  implement  linear  or  quadratic  discriminant  functions. 
The  outcome  of  this  algorithm  is  an  initial  approximation  of  a  suitable  network.  We 
have  tested  it  with  Fisher's  iris  data  set  (Fisher,  1936).  Fisher's  data  set  has  four 
variables  for  three  different  kinds  of  iris's.  Using  Jackknife  resampling  methods  (Efron, 
1982),  average  overall  cross-validation  classification  performance  was  0.973  while 
average  performance  on  the  data  used  to  set  up  the  networks  for  each  Jackknife  sample 
was  0.98.  These  results  are  comparable  to  results  reported  for  statistical  classification 
methods  as  well  as  common  neural  network  algorithms  (Weiss  and  Kulikowski,  1991). 

The  next  step  in  this  development  is  to  implement  algorithms  to  refine  the  network  by 
adjusting  connection  weights  between  neural  units,  automatically  add  neural  units  to  the 
network  if  needed,  and  finding  alternative  combinations  of  variables  that  have  higher 
classification  power.  The  final  step  will  be  to  convert  network  outputs  into  a  continuous 
variable  and  to  demonstrate  the  feasibility  of  using  it  as  a  continuous  index  of  mental 
workload.  No  revisions  to  the  techniques  described  in  the  Phase  II  proposal  are 
anticipated  at  this  time. 

Experiment  for  Testing  the  Algorithm 

Design  of  the  Experiment 

The  experiment  consists  of  recording  EEGs,  eye  movement  and  blink  potentials,  and 
EKG  while  subjects  perform  a  battery  of  neurocognitive  tasks.  The  task  battery  is 
composed  of  several  task  families  which  differ  in  the  variety  and/or  degree  of  mental 
resources  involved.  Component  tasks  within  each  family  differ,  by  design,  in  mental 
workload;  verification  is  by  performance  variables  and  the  Subjective  Workload 
Assessment  Test  (SWAT).  These  data  are  used  to  find  electrophysiological  indices  of 
mental  workload  that  work  across  task  families  and  subjects.  Performance  (reaction 
time  and  accuracy)  and  SWAT  scores  are  used  to  separate  tasks  into  four  levels  of 
mental  workload.  Using  data  from  three  of  these  levels,  the  network  algorithm  is  used 
to  construct  a  network  that  implements  a  continuous  index  of  mental  workload.  The 
network  is  then  tested  on  its  ability  to  place  the  data  from  the  excluded  level  within  an 
expected  interval  of  the  index.  This  analysis  is  performed  with  two  subsets  of  task 
families  to  test  generalizability  of  the  index  across  tasks  and  with  two  subsets  of  subjects 
to  test  generalizability  across  subjects;  Jackknife  resampling  methods  (Efron,  1982)  are 
used  to  obtain  improved  estimates  of  generalizability  performance. 

Each  subject  performs  the  whole  battery  of  tasks  twice,  using  a  different  randomized 
order  each  time.  During  the  first  run,  no  electrophysiological  recordings  are  made  and, 
for  each  task,  additional  task  trials  arc  given  until  subjects  respond  correctly  at  least 
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80%  of  the  time,  indicating  that  they  have  understood  the  requirements  of  the  task. 
During  the  second  run,  16-32  channels  of  EEG  at  the  standard  10-20  positions  are 
recorded  continuously  throughout  each  task  with  a  passband  of  0.05  to  50  Hz  and 
referenced  to  digitally  linked  mastoids.  Fifteen  trials  or  less  are  given  for  each  task 
with  maximum  play  time  on  any  one  task  limited  to  2  minutes.  After  each  task,  subjects 
evaluate  their  experience  on  the  task  by  performing  the  SWAT. 

Cognitive  Performance  Tasks 

Our  strategy  is  to  test  the  workload  classification  algorithm  for  generalizability  across 
tasks  by  designing  several  types  of  tasks  that  each  vary  in  the  degree  of  workload  that 
they  impose  and  the  type  of  cognitive  operations  that  they  require.  The  mental  workload 
index  is  then  tested  for  generalizability  across  tasks,  providing  a  strong  test  that  the 
index  is  not  peculiar  to  a  particular  task  family  or  type  of  cognitive  operation.  In 
designing  the  battery  an  effort  was  made  to  explicitly  test  cognitive  functions  relevant  to 
piloting  an  aircraft,  including: 

Working  Memory— maintaining  information  for  several  seconds  while  it  is  being  utilized 
in  the  context  of  a  mental  task; 

Visuomotor  Tracking-sustained  execution  of  adaptive  actions  in  response  to  some 
continuously  varying  perceptual  event; 

Dynamic  Visuospatial  Reasoning—Tzpidiy  coordinating  behavior  with  information  about 
the  location  and  speed  of  moving  objects;  and. 

Divided  >i/renrio/i— simultaneously  allocating  attention  to  multiple  tasks. 

Each  of  these  types  of  tasks  have  been  studied  in  detail  (Pelligrino  &  Hunt,  1990; 
Baddeley,  1992;  Poulton,  1974;  Wickens,  1984).  Our  testing  protocol  examines  the 
physiological  concomitants  of  increases  in  the  workload  imposed  on  each  of  these  types 
of  operations. 

Task  Battery 

For  each  of  the  mental  faculties  listed  above,  except  for  Dynamic  Visuospatial 
Reasoning,  we  have  specified  two  task  families  which  differ  in  the  form  of  the 
information  processed;  in  one,  spatial  information  must  be  processed  while  the  other 
requires  symbolic  processing.  The  tasks  are  presented  visually  on  a  computer  screen 
and  require  speeded  finger-press  responding. 

Working  Memory 

For  this  category,  the  basic  idea  is  to  present  stimuli  in  a  continuous  sequence  and 
require  the  subject  to  compare  recent  stimuli  with  those  earlier  in  the  sequence.  In  the 
family  of  tasks  oriented  toward  symbolic  processing,  the  task  is  to  determine  whether  or 
not  a  stimulus  object  matches  the  previous  one.  For  the  next  higher  mental  workload 
level,  the  task  is  to  compare  the  present  stimulus  with  the  second  prior  stimulus.  The 
third  level  will  require  comparing  the  prior  stimulus  with  the  second  prior  stimulus.  The 
fourth  level  will  be  the  same  as  the  second  level  but  stimuli  will  be  presented  at  a  faster 
pace.  The  task  family  oriented  toward  spatial  processing  will  be  the  same  except  that 
the  task  will  be  to  determine  whether  or  not  stimuli  appear^  in  the  same  spatial  location 
on  a  circle  (Mishkin  et  al.,  1983). 
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Visuomotor  Tracking 

For  the  task  family  oriented  toward  spatial  processing,  the  task  is  to  keep  a  horizontally 
moving  cursor  centered  on  the  computer  screen  using  continuous  controllers  in  the  right 
and  left  hands  to  influence  right  and  left  cursor  movement  respectively  (Jex,  1966). 
Cursor  motion  reflects  the  output  of  an  unstable  control  system  characterized  by  a 
transfer  function  with  one  or  more  poles,  gain,  and  an  instability  parameter,  the  control 
system  tends  to  amplify  any  deviation  from  the  central  cursor  position.  The  first  three 
mental  workload  levels  correspond  to  one-,  two-,  and  three-pole  systems  respectively. 
The  instability  parameter  is  gradually  increased  until  the  operator  loses  control  and  the 
cursor  exits  one  side  of  the  screen.  The  fourth  workload  level  is  the  same  as  the  second 
level  except  that  the  cursor  also  exhibits  vertical  deviations  from  center;  vertical 
deviations  must  be  nullified  by  a  right  or  left  button  press  for  up  or  down  motion 
respectively.  For  the  task  family  oriented  toward  symbolic  processing,  cursor  position 
is  given  by  a  numerical  display  (or  two  in  the  case  of  the  fourth  level)  instead  of 
graphically.  The  task  is  to  keep  the  read-out  at  zero  as  closely  as  possible. 

Dynamic  Visuospatial  Reasoning 

The  basic  idea  for  the  task  family  in  this  category  is  to  launch  a  "missile",  which  appears 
on  the  bottom  of  the  computer  screen,  in  such  a  way  that  it  intercepts  a  target  object 
traveling  horizontally  across  the  computer  screen  (Hunt  et  ah,  1988).  The  venical 
position  of  the  target  differs  randomly  from  trial  to  trial  and  the  speed  and  trajectory 
differs  for  different  mental  workload  levels.  The  "missile"  travels  a  linear  trajectory. 
Mental  workload  is  increased  by  increasing  the  number  of  parameters  that  must  be 
accurately  controlled  for  successful  interception.  For  the  first  level,  only  launch  time 
need  be  controlled.  At  the  second  level,  launch  force  must  also  be  controlled.  At  the 
third  level,  horizontal  launch  position  must  also  be  controlled.  At  the  fourth  level,  the 
slope  of  the  launch  trajectory  must  also  be  controlled.  There  is  no  symbolic  version  of 
this  task. 


Divided  Attention 

The  basic  idea  for  this  category  is  to  perform  several  tasks  at  once,  each  of  which 
emphasize  a  different  collection  of  mental  resources  (Wickens,  1984).  The  first  mental 
workload  level  consists  of  two  tasks,  increasing  to  five  tasks  for  the  fourth  level.  The 
component  tasks  of  the  spatially  oriented  task  family  includes  the  following:  (1)  a  tone 
detection  task;  (2)  the  lowest  level  spatially  oriented  Working  Memory  task;  (3)  a 
simplified  form  of  the  spatially  oriented  Visuomotor  Tracking  task  where  a  button  press 
simply  centers  the  horizontally  moving  cursor;  (4)  the  lowest  level  Dynamic 
Visuospatial  Reasoning  task;  and  (5)  an  arithmetic  task  adapted  from  the  NATO 
AGARD  STRES  battery  (AGARD,  1989)  In  the  tone  detection  task,  the  computer 
periodically  emits  a  tone  to  either  the  left  or  right  ear,  one  of  which  occurs  less  often. 
The  goal  is  to  respond  with  a  button  press  when  the  rare  tone  (side)  occurs.  In  the 
arithmetic  task,  the  sum  or  difference  of  two  graphical  columns  is  to  be  judged  taller  or 
shorter  than  a  reference  column.  The  component  tasks  of  the  symbolically  oriented 
task  family  includes  the  following:  (1)  the  tone  detection  task  where  the  tones  can  be  a 
high  or  a  low  sound  delivered  to  both  ears;  (2)  the  lowest  level  symbolically  oriented 
Working  Memory  task;  (3)  the  symbolic  version  of  the  Visuomotor  Tracking  task, 
simplified  in  the  manner  described  above;  (4)  a  symbolic  version  of  the  lowest  level 
Dynamic  Visuospatial  Reasoning  task;  and  (5)  the  arithmetic  task  with  numbers  in 
place  of  the  graphical  columns.  In  the  symbolic  version  of  ±e  lowest  level  Dynamic 
Visuospatial  Reasoning  task,  a  numerical  counter  is  presented  along  with  a  target 
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